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ABSTRACT 

The 1990 Carl D. Perkins Vocational and Applied 
Technology Education Act established an accountability system for 
vocational education based on "outcomes" such as academic skill 
gains, job placement, and program completion. Current congressional 
proposals for workforce training differ in the emphcsis they place on 
individuals, communities, and states in the accountability process, 
and some proposals could result in abandonment of the Perkins Act's 
outcome-based model for program accountability and improvement before 
its effectiveness has been fully evaluated. The current group of 
federal legislative proposals for workforce training do not give 
sufficient attention to which outcomes should be monitored, and most 
proposals are vague about the use of outcome data. Research examining 
the broad issue of accountability and the effects of Perkins Act 
provisions on workoforce training in particular has suggested the 
following conclusions that are germane to the congressional debate 
over accountability: (1) workforce training programs should be 
accountable to multiple co.istituents (students, the local business 
community, and the state); (2) th<s choice of outcomes monitored 
affects the way an educational system operates and can distort system 
performance; and (3) producing performance data does not guarantee 
that such data will be used effectively. (MN) 
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Congress may soon make major changes in federal 
workforce training programs. However, the proposals 
being considered are vague about accountability: to 
whom should programs be accountable, which outcomes 
should be monitored, and how should performance data 
be used? Research carried out by the National Center for 
Research in Vocational Education (NCRVE) has been 
exploring these qui stions in the context of vocational edu- 
cation for the last several years. The research suggests 
that the way these questions are answered will affect the 
delivery of training in significant ways. 

THE CHANGING POLICY CONTEXT 

The 1990 Carl D. Perkins Vocational and Applied 
Technology Education Act (known as Perkins II) estab- 
lished an accountability system for vocational education 
based on "outcomes" — factors such as academic skill 
gains, job placement, and program compl .ion. States 
were required to establish systems of standards and mea- 
sures of performance that local programs could use as a 
basis for program improvement. For the past four years, 
the states have been developing an information infrastruc- 
ture to provide the appropriate outcome data. Vocational 
educators and policymakers have high expectations that 
vocational training programs will improve as a result. 

However, we may never learn if this outcome-based 
model for program improvement works, because mem- 
bers of Congress are likely to abandon Perkins II before 
these systems are fully operational. Current congressional 



proposals would reduce the federal government's role by 
consolidating job training efforts into a smaller number of 
programs that have fewer prescriptive guidelines. The 
focus of recent legislative initiatives is on shifting authori- 
ty to the states and reducing costs; relatively little atten- 
tion is given to accountability. The risk is that workforce 
training nationwide may suffer if neither the federal gov- 
ernment nor the states continue to develop accountability 
mechanisms. 

TO WHOM SHOULD PROGRAMS BE 
ACCOUNTABLE? 

Current legislative proposals for workforce training 
differ in the emphasis they place on individuals, commu- 
nities, and states in the accountability process. Since these 
three institutional "actors" can have different goals, their 
relative role in accountability can change the way the sys- 
tem behaves. At one extreme, a voucher approach makes 
individual participants the agents of accountability. If 
participants are not satisfied with a program or provider, 
they can "vote with their feet" by taking their training 
vouchers elsewhere. At the other extreme, many block 
grant proposals give state agencies the responsibility for 
insuring quality and protecting students from unsatisfac- 
tory training programs. When state agencies hold the 
accountability reins, they vote with their dollars by termi- 
nating poor-quality programs. An intermediate approach 
assigns decisionmaking authority to local community 
councils who, in theory, are more responsive to the needs 
of the local business community. Under these conditions. 



accountability reflects a negotiated balance between the 
needs of employers, providers, and students. 

Placing the responsibility for program accountability 
with individuals, states, or local communities will have 
different effects on programs because these three groups 
have different goals. Individual students place greater 
emphasis on their personal employment goals; states tend 
to be more respectful of the concerns of schools and dis- 
tricts; and local communities are more responsive to the 
needs of local employers. When personal goals, institu- 
tional goals, and employers' goals conflict, as they often 
do, the assignment of responsibility for accountability to 
one group or another will make a difference. For exam- 
ple, students' personal employment goals can be at odds 
with long-term community or state goals for the work- 
force. Administrators in one school studied by NCRVE 
researchers terminated a child care worker training pro- 
gram with high enrollment because the program led 
women into traditionally low-paying jobs with little 
chance for advancement. The women enrolled in the pro- 
gram may not have made the same choice. 

There are other consequences of assigning account- 
ability to individuals or states. Voucher holders with 
information about program quality will exercise their 
accountability functions (by enrolling or withdrawing) 
relati\'ely quickly and decisively; in contrast, states can 
take much longer to terminate a marginal program. 
Schools tend to defend their teachers and programs, and 
states are reluctant to take decisi\ e actions against 
schools, especially if they belie\ e that improvements may 
be possible. On tlie other hand, if accountability is vested 
in individual voucher holders, the pressure to eliminate 
programs may hold sway over the pressure to improve 
them. 

WHICH OUTCOMES SHOULD BE MONITORED? 

Tlie current crop of federal legislative proposals for 
workforce training gives insufficient attention to which 
outcomes should be monitored. Regardless of who is 
responsible for monitoring quality, valid measures of per- 
formance must be identified. The decision about which 
outcorr 'S should be given priority — labor force outcomes, 
occupauona2 skills, or program completion — affects the 
way the system operates and can create incentives for 
behaviors that threaten the quality of the training. 

Since the goal of job training programs is employ- 
ment, the most n.ntural outcomes to monitor are tliose 
associated with entry into the labor force. Labor force out- 
comes include initial employment, wage levels, continu- 
ing employment, advancement, and employee and 



employer satisfaction. However, building accountability 
around employment can lead to ir.valid conclusions and 
incorrect actions. For example, fluctuations in the local 
labor market pose a difflcult measurement problem. 
Declines in placements might not indicate program failure 
so much as an economic downturn. Emphasizing place- 
ment as a measure of program success may also affect: 

• Who is selected for training — those perceived to be 
most "employable" or those most in need 

• Which occupations are the focus of training — thc-e 
that are easiest to "train to" or those demanding the 
most difficult learning 

• Which skills are emphasized— those that satisfy 
employers' initial hiring demands or those that bolster 
employees' long-term career potential. 

The long-term health of local and state economies 
might be better served by a system that focuses on skills, 
rather than labor market success, as the essential outcome. 
In fact, the recent development of national skill standards 
establishes a framework that could be used as such a basis 
for accountability. In the long run, economic productivity 
might be enhanced more by emphasizing general work- 
force skills and competencies, such as time management, 
teamvi'ork, and understanding of technologies and sys- 
tems, rather than initial labor market outcomes. 
Furtherm>ore, defining success in terms of skill impro\'e- 
ment rather than skill attainment may create incentives to 
serve different students. For example, if improvement is 
rewarded, programs may emphasize service to those with 
the lowest skills, belicrving they have the greatest potential 
to grow. In contrast, if attainment is rewarded, those with 
the highest skills may appear more attractive because they 
are more likely to meet completion criteria. 

Another set of performance outcomes can be defined 
in terms of program participation. Useful measures 
include program and course enrollment, continuation, 
and com.pletion. Focusing on program participation pro- 
vides more immediate information about the quality of 

ssroom services and increases the likelihood that pro- 
gram deflciencies can be identified ard improved. It also 
makes it easier to address concerns about equity of access 
and services. In the long run, the workforce preparedness 
system is only as good as its programs, so attention to 
program quality is important. At the same time, pertici- 
pants' goals do not always match program goals, and 
measuring outcomes only in terms of program completion 
can lead to incorrect inferences, as well. In particular, 
some students learn the skills they need to And employ- 
ment before they complete the coherent set of courses that 
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define a program. These "non-completers" who leave to 
take a job may be satisfied with the outcome even though 
they have not attained the desired result — program com- 
pletion — from the point of view of the program. 

It is not easy to say which outcomes are the right ones 
to monitor. Perkins II encourages states to include many 
outcomes rather than focusing narrowly on employment, 
skills, or program participation alone. Given some flexibil- 
ity in their choices, states have opted to include more 
rather than fewer measures. Assessment professionals 
strongly eiidorse the use of multiple measures that reflect 
the objectives of the program. 

HOW SHOULD OUTCOME DATA BE USED? 

The authors of Perkins II believed that students would 
be served best if performance outcomes were used to make 
programs more responsive to student needs. This empha- 
sis on local responsibility for program improvement was 
one of the distinguishing characteristics of the legislation. 
Only if programs were unable to improve themselves did 
states step in — to offer guidance or, if necessary, to termi- 
nate the program. The current legislative proposals for 
workforce training are vague abont the use of outcome 
data, and some even support a punitive approach. They 
assume tha' outcome data are to be used by the state to 
assure program quality; if the data show deficiencies, the 
state can revoke .'unding. In failing to mention program 
improvement, however, congressional proposals are over- 
looking one of the most potent uses of performance data. 
And they ignore the fact that, in practice, programs are 
rarely terminated. The costs — both political and econom- 
ic — are simply too great. This dichotomous approach — to 
fund or not to fund — may be satisfying on a visceral level, 
but it does not constitute good policy. 

NCRVE research has found that local accountability 
systems can be effective tools for program improvement. 
In tact, recent research recommends making a broader 
scope of information available to local decisionmakers so 
that they can become iigents of ref i. itcome measures 



alone do not provide information about the causes of prob- 
lems — only about their effects. Information about why 
programs are failing to meet their goals is far more useful 
for purposes of reform than counts of completers and 
placements. 

The research also calls for increasing the expertise of 
people who are selecting, collecting, and analyzing data 
and then using the information for program decisions. 
Studies of school reform have shown that data collection is 
seldom a catalyst for change, particularly mandated data 
collection. Schools tend to use data only to signal their 
compliance with regulations, not as the basis for informed 
program improvement. And vocational educators, like 
their counterparts in general education, have limited expe- 
rience with the use of data to manage or improve pro- 
grams. 

MAKING ACCOUNTABILITY WORK 

Research conducted by NCRVE has been examinirig 
the broad issue of accountability, and specifically the 
effects of the provisions of Perk ;is II on workforce train- 
ing, for the last four years. As "suggested here, three con- 
clusions are germane to the congressional debate. First, 
workforce training programs should be accountable to 
multiple constituents — students, the local business com- 
munity, and the state. Shifting the emphasis to a single 
constituency creates unbalanced incentives that could 
undermine the quality of training. Second, it matters a 
great deal which program outcomes are monitored — labor 
force outcomes, occupational skills, or program comple- 
tion. The choice of outcomes affects the way the system 
operates and can distort system performance. Multiple 
outcomes measures as well as data about instructional pro- 
cesses are preferred. Third, producing performance data 
does not guarantee that such data will be used effectively. 
Provisions must be made to help state and local authorities 
and program administrators use information in the most 
effective way: that is, to make beneficial choices, to 
strengthen successful training programs, and to eliminate 
unsuccessful ones. 
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